The present invention relates to techniques for evaluating the performance of binary classification systems. More particularly, specific embodiments of the invention relate to fraud detection systems.
“Click-based” online advertising systems require an advertiser to pay the system operator or its partners each time a user selects or “clicks” on the advertiser's online advertisement or sponsored search link. Unfortunately, the nature of such a system provides opportunities for some to click on ads for improper or fraudulent reasons. This is referred to generally as “click fraud.” For example, a provider of online advertising services may partner with a third party to place ads for an advertiser on the third party's web site with a portion of the revenue for each click going to the third party. This provides a financial incentive for the third party to click the links on its own site. In another example, one company might be motivated to click on the ads of a competitor to drive up advertising costs for the competitor. Some click fraud efforts are fairly large in scale with groups of people being paid to engage in such activity, i.e., “click farms.” There are even automated process for engaging in click fraud, e.g., web crawling bots, ad-ware, and various kinds of mal-ware.
The rapid rise in click-based online advertising, and the ease with which click fraud may be perpetrated has spurred the development of systems designed to detect click fraud. Such systems evaluate click events with reference to one or more of a wide range of criteria to determine whether a click is “good,” e.g., a valid click by an interested consumer, or “bad,” i.e., a fraudulent click. For example, clicks by self-declared bots may be automatically identified as fraudulent. In addition, a large number of clicks from the same user within a specified period of time may be identified as fraudulent. The clicks are then filtered on this basis and the advertisers billed accordingly.
Unfortunately, given the difficulty in determining whether a click event amounts to click fraud, click fraud detection systems typically generate some number of false positives, i.e., valid events which are incorrectly identified as invalid or fraudulent, and false negatives, i.e., invalid or fraudulent events which are incorrectly identified as valid. In addition, it is extremely difficult to evaluate the performance of a click fraud detection system in that it is difficult, if not impossible, to determine the number of false negatives. That is, a false negative is difficult to identify because there is no evidence that the click event identified as valid is fraudulent, i.e., it is indistinguishable from many other valid click events.
Thus, because it is nearly impossible to distinguish false negatives from valid events, it is extremely difficult to evaluate the performance of click fraud detection systems. This is problematic in that it undermines advertisers' confidence that they are paying for valid events.